This quick test is showing -muge giving a boost +6.8% at short kv-cache depth and +2.8% faster near 128k depth with my Qwen3.5-122B-A10B-IQ4_KSS. Its been running well doing some light testing today including mmproj. 🚀

model=/ubergarm/Qwen3.5-122B-A10B-GGUF/Qwen3.5-122B-A10B-IQ4_KSS.gguf
./build/bin/llama-sweep-bench \
  --model "$model" \
  -c 135168 \
  -sm graph \
  -ngl 999 \
  -ub 4096 -b 4096 \
  --threads 1 \
  --no-mmap \
  -n 128 \
  --warmup-batch

model=/ubergarm/Qwen3.5-122B-A10B-GGUF/Qwen3.5-122B-A10B-IQ4_KSS.gguf
./build/bin/llama-sweep-bench \
  --model "$model" \
  -c 135168 \
  -sm graph \
  -muge \
  -ngl 999 \
  -ub 4096 -b 4096 \
  --threads 1 \
  --no-mmap \
  -n 128 \
  --warmup-batch

I don't have a mainline compatible quant handy to test on this rig, but have seen that issue where sweep-bench fails towards the end on mainline too as you mentioned in another PR.

Is on-the-fly merged ffn_up/gate_exps ggufs faster than just using --merge-up-gate-experts on a non-merged ggufs?

Is on-the-fly merged ffn_up/gate_exps ggufs faster than just using --merge-up-gate-experts on a non-merged ggufs?

It is the same thing. I started using "on-the-fly merged" for --merge-up-gate-experts to distinguish from the case where these tensors have already been merged in the model stored on disk (see issue #1399 that you entered yourself)

I'm confused. Is that means using pre-merged ggufs is same as using non-merged ggufs with option -muge?

I'm confused. Is that means using pre-merged ggufs is same as using non-merged ggufs with option -muge?

Yes, it is the same. In the pre-merged case, someone (for instance AesSedai) has prepared the model such that the ffn_up_exps and ffn_gate_exps tensors are merged into a single ffn_gate_up_exps tensor and stored the model that way on disk. In that case, when we load the model we don't need to do anything to take advantage of the merge. With the on-the-fly merge (-muge) the model stored on disk contains separate ffn_up_exps and ffn_gate_exps tensors, and we merge them on-the-fly while loading the model. The end result (i.e., what happens during inference) is exactly the same.

In my experience, running -sm graph -muge for hybrid GPU+CPU inference (with tensor offloading) of Qwen 3.5 397B IQ4_KSS from ubergarm makes the output fall into loops. Flags used:

      -c 210000 \
      --jinja \
      -fa 1 -ngl 99 -ub 4096 -b 8192 \
      --ctx-checkpoints 12 --ctx-checkpoints-interval 16383 \
      -cuda fusion=1,offload-batch-size=4,mmq-id-size=128 \
      -gr -ger \
      --split-mode graph -smgs --graph-reduce-type f32 \
      -muge \
      -ts 35,26 \
      -ot "blk\.([0-2])\.ffn_(up|gate|down)_exps\.weight=CUDA0" \
      -ot "blk\.([5][7-9])\.ffn_(up|gate|down)_exps\.weight=CUDA1" \
      -ot "blk\.([0-9]|[1-9][0-9])\.ffn_(up|gate|down)_exps\.weight=CPU" \
      --no-warmup --no-mmap

If I don't use -muge I get proper outputs.

I suppose it is due to the "unexpected results if using custom tensor offloads with split-mode graph" warning. I am thankful that it still works very well without merging up and gate expert tensors, so this is just a drawback of using custom tensor offloading.

@abc-nix

I'm not 100% sure, but digging through some recent PRs on imatrix fused up|gate tensors and the original -muge PR it may be that when you use -muge that your -ot needs to change from ffn_(up|gate|down)_exps to be ffn_(gate_up|down)_exps ... might give that a try?

Same thing if you're using a mainline pre-merged quant...

ik was gracious and re-named the existing convention here to reduce confusion with the new opposite naming convention on mainlin...

so it is ffn_gate_up_exps everywhere now psure, gonna go submit a PR to add this to --cpu-moe and --n-cpu-moe regex strings now

@ubergarm, thanks for the tips. This will help when using models with these experts merged (I will be using a catch-all regex ffn_(gate_up|up|gate|down)_exps).

What I am seeing when offloading the up and gate tensors with a non-merged gguf (the one in your repo), is that they are merged after they are loaded to the device. If I use -ot "blk\.([0-2])\.ffn_(gate_up|up|gate|down)_exps\.weight=CUDA0", I can see:

Tensor blk.0.ffn_up_exps.weight (size = 1026.00 MiB) buffer type overriden to CUDA0
Tensor blk.0.ffn_gate_exps.weight (size = 1026.00 MiB) buffer type overriden to CUDA0
merge_up_gate_exps: merging up/gate in layer 0
Tensor blk.0.ffn_down_exps.weight (size = 1096.00 MiB) buffer type overriden to CUDA0
Tensor blk.1.ffn_up_exps.weight (size = 1026.00 MiB) buffer type overriden to CUDA0
Tensor blk.1.ffn_gate_exps.weight (size = 1026.00 MiB) buffer type overriden to CUDA0
merge_up_gate_exps: merging up/gate in layer 1
Tensor blk.1.ffn_down_exps.weight (size = 1096.00 MiB) buffer type overriden to CUDA0
Tensor blk.2.ffn_up_exps.weight (size = 1026.00 MiB) buffer type overriden to CUDA0
Tensor blk.2.ffn_gate_exps.weight (size = 1026.00 MiB) buffer type overriden to CUDA0
merge_up_gate_exps: merging up/gate in layer 2
Tensor blk.2.ffn_down_exps.weight (size = 1096.00 MiB) buffer type overriden to CUDA0
[...]

So that is not the issue. The output still repeats itself, so using -muge with -sm graph and partial expert offloading is bad on my machine (unless there is a conflict with a different flag).